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Program Doscription^ The university of Chicago Schooi Mathematics Project (UCSMP) 

6-12 Curricuium is a series of yearlong courses— (1) Transition 
Mathematics; (2) Algebra; (3) Geometry; (4) Advanced Algebra; 

(5) Functions, Statistics, and Trigonometry; and (6) Precalculus 
and Discrete Mathematics— emphasizing problem solving, 
real-world applications, and the use of technology. The program 



is designed to allow schools to offer the appropriate math to 
students regardless of grade level. Beginning with the Algebra 
course, technology is used in the classroom to aid in the devel- 
opment of properties and skills, and graphing calculators are 
used to complete assignments at all levels. 



Research^ Two studies of UCSMP that fall within the scope of the High School 
Math review protocol meet What Works Clearinghouse (WWC) 
evidence standards with reservations. Together, the two studies 
included 251 high school students in five schools in five districts.® 



Based on these two studies, the WWC considers the extent 
of evidence for UCSMP on high school students to be small for 
math achievement because the ratings are based on reviews 
with fewer than 350 students in total. 



EffactivanaSS UCSMP was found to have potentially positive effects on math achievement for high school students. 

Math achievement 

Rating of effectiveness Potentially positive 

Improvement index"^ Average: +23 percentile points 

Range: +2 to +38 percentile points 

1. The descriptive information for this program was obtained from publicly available sources: the developer’s website (http://ucsmp.uchicago.edu, down- 
loaded April 2010) and the publisher’s website (http://www.mheonline.com, downloaded August 2010). The WWC requests developers to review the 
program description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive information for this program is 
beyond the scope of this review. The literature search reflects documents publicly available by October 2010. 

2. The studies in this report were reviewed using WWC Evidence Standards, Version 2.0 (see the WWC Procedures and Standards Handbook, Chapter III), 
as described in protocol Version 2.0. 

3. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

4. These numbers show the average and range of student-level improvement indices for all findings across the studies. 



WWC Intervention Report University of Chicago School Mathematics Project 6-12 Curriculum 



July 2011 







Additional program 
information 



Research 



Developer and contact 

Developed by the University of Chicago School Mathematics 
Project (UCSMP), the third edition of the University of Chicago 
School Mathematics Project 6-12 Curriculum is distributed 
by Wright Group/McGraw-Hill. Email: SEG_customerservice@ 
mcgraw-hill.com. Web: http://www.mheonline.com. Telephone: 
(800) 523-2371. 

For additional information on the program, contact either 
Zalman Usiskin, professor emeritus of education and director, 
UCSMP (Address: 6030 South Ellis Avenue, Chicago, IL 60637. 
Email: z-usiskin@uchicago.edu. Telephone: (773) 702-1560), 
or Denisse Thompson, professor of mathematics education. 
University of South Florida (Address: 4202 East Fowler Ave., 
#DU105, Tampa, FL 33620-5650. Email: denisse@usf.edu. Tele- 
phone: (813) 974-2687). 

Scope of use 

The first edition of UCSMP was developed and tested beginning 
in 1985 and the second edition beginning in 1992.® The third 
edition was developed and tested between 2005 and 2007 and 
is available through the Wright Group/McGraw-Hill. According to 
the developers at the University of Chicago School Mathematics 
Project, more than 3.5 million students in elementary, middle, 
and high schools are using UCSMP materials and curricula. 



Teaching 

Each course of the UCSMP includes the following components: 
the student textbook (hardcover), teacher’s edition (including the 
electronic teacher’s edition, eTe), teacher resources, assessment 
resources, and technology resources. Lessons in the student 
book contain activities, full examples, and partially completed 
guided examples to model skills and problem solving. Students 
are encouraged to assess their own understanding with an End- 
of-Chapter Self-Test correlated to objectives. Projects provided 
at the end of each chapter are designed as extended activities, 
giving students experience using real data. The use of technol- 
ogy-including graphing calculators at all grade levels, geometry 
systems, spreadsheets, the Internet, and other computer appli- 
cations— is considered an essential component of the curricula. 

Cost 

For each individual curriculum, the student textbook costs 
$63.00. A bundled, complete teacher resource package 
consisting of the teacher’s edition and teacher resources (vol- 
umes 1 and 2), assessment CD-ROM, and electronic teacher’s 
edition with answers and solutions (volumes 1 and 2) costs 
$346.50. See the publisher’s website for pricing of individual 
resource items. 



Twenty studies reviewed by the WWC Fligh School Math topic 
area investigated the effects of any of the six topic areas of 
UCSMP. Two studies (Hirschhorn, 1993; Thompson, Senk, 
Witonsky, Usiskin, & Kaeley, 2006) are quasi-experimental 
designs that meet WWC evidence standards with reservations. 
The remaining 18 studies do not meet either WWC evidence 
standards or eligibility screens. 



Meets evidence standards with reservations 

Flirschhorn (1993) conducted a longitudinal, four-year quasi- 
experimental evaluation of the UCSMP (first edition) in three high 
schools in which both traditional and UCSMP curricula were 
used. Every student in the intervention group received four years 
of UCSMP curricula, starting with Transition Mathematics in 7th 
grade and ending with L/CSMP Advanced Algebra in 10th grade. 
No student in the comparison group participated in any UCSMP 
courses, instead enrolling in math classes using other curricula. 



5. The two studies meeting standards with reservations in this report focus on different editions of the program. Per Thompson et al. (2006), the major 
differences between the first and second editions are an increased use of technology and emphasis on student writing and projects. 
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Research (continued) Treatment students were matched to potential comparison 

students retrospectively using pretest (6th-grade) scores, and 
outcomes were measured at the end of the four-year period (10th 
grade). Two high schools (B and C in the study), located in affluent 
suburbs with predominantly White, higher-achieving students, 
contributed a total of 62 students to the analysis. A third high 
school (A in the study) is excluded from the WWC review because 
baseline equivalence between the intervention and comparison 
groups could not be established on math scores alone. 

In the second study meeting evidence standards with reserva- 
tions, Thompson et al. (2006) used students’ math ability to match 
pairs of classrooms within a school to evaluate the effectiveness 
of the second edition of UCSMP Algebra. Random assignment of 
teachers, within each pair, to the treatment or control condition 
was not always possible; therefore, this study is treated as a 
quasi-experimental design. The WWC review focuses on the three 



high schools (X, Y, and Z in the study) that compared the use of the 
UCSMP second edition to that of non-UCSMP textbooks.® These 
three high schools, located in three different regions of the country 
and serving three different populations of students, contributed 189 
primarily 9th-grade students to the analysis. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or medium to large (see the WWC Procedures and Standards 
Handbook, Appendix G). The extent of evidence takes into account 
the number of studies and the total sample size across the studies 
that meet WWC evidence standards with or without reservations.^ 
The WWC considers the extent of evidence for UCSMP to be 
small for math achievement for high school students because 
the ratings are based on reviews with fewer than 350 students in 
total (see Appendix A5). 



Effectiveness Findings 

The WWC review of interventions for High School Math 
addresses student outcomes in one domain: math achievement. 
The findings below present the authors’ estimates and WWC- 
calculated estimates of the size and the statistical significance of 
the effects of UCSMP on high school students.® 

Math achievement. Two studies presented findings in the 
math achievement domain. Using the Mathematics Level 1 
Achievement test, Hirschhorn (1993) reports no statistically 
significant effect in one school, and a positive and statistically 
significant effect in another. WWC calculations pooled samples 



across the two sites and found differences that were neither sta- 
tistically significant (at the 0.05 level) nor large enough to be con- 
sidered substantively important according to WWC criteria (i.e., 
an effect size of at least 0.25). Based on a second measure, the 
UCSMP-developed Applications Test, Hirschhorn (1993) reports, 
and the WWC confirms, positive and statistically significant 
effects. Thompson et al. (2006) report no statistically significant 
effects on the standardized High School Subjects Test: Algebra, 
but positive and statistically significant effects on both the 
UCSMP-developed Algebra® and Problem-Solving tests. The 
WWC confirmed the statistical significance of the latter findings. 



6. Another analysis included schools in which the comparison group used UCSMP Algebra, first edition. This design is not eligible for review based on the 
topic area review protocol. 

7. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for UCSMP is in Appendix A5. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of 
Hirschhorn (1993), a correction for multiple comparisons was needed, so the significance levels may differ from those reported in the original study. In 
the case of Thompson et al. (2006), corrections for clustering and multiple comparisons were needed, so the significance levels may differ from those 
reported in the original study. 

9. Care must be taken in interpreting results for the UCSMP-developed Algebra test, as comparison students in school Y had an opportunity to learn the 
content for less than one-quarter of the test items. 
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Effectiveness (continued) 



References 



using clustering and muitiple comparison corrections. In sum, 
both studies demonstrated statisticaiiy significant positive 
effects in the math achievement domain, using smali sampies 
of students and oniy on developer-created instruments.''° 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentialiy positive, mixed, no discernibie 
effects, potentiaiiy negative, or negative. The rating of effective- 
ness takes into account four factors: the quaiity of the research 
design, the statistical significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Procedures and Standards Handbook, 
Appendix E). 

The WWC found UCSMP to have potentially positive effects 
on math achievement for high schooi students. 

Improvement index 

The WWC computes an improvement index for each individuai 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 



average improvement index across studies (see WWC Procedures 
and Standards Handbook, Appendix F). The improvement index 
represents the difference between the percentiie rank of the aver- 
age student in the intervention condition and the percentiie rank of 
the average student in the comparison condition. Unlike the rating 
of effectiveness, the improvement index is entirely based on the 
size of the effect, regardless of the statistical significance of the 
effect, the study design, or the analysis. The improvement index 
can take on vaiues between -50 and +50, with positive numbers 
denoting favorabie resuits for the intervention group. 

The average improvement index for math achievement is +23 
percentiie points across the two studies, with a range of +2 to 
+38 percentile points across findings. 

Summary 

The WWC High School Math topic area reviewed 20 studies on 
the UCSMP. Two of these studies meet WWC evidence stan- 
dards with reservations; the remaining 18 studies do not meet 
either WWC evidence standards or eiigibiiity screens. Based 
on the two studies, the WWC found potentially positive effects 
in math achievement for high schooi students. The conclusions 
presented in this report may change as new research emerges. 



Meets WWC evidence standards with reservations 

Hirschhorn, D. B. (1993). A iongitudinal study of students 
completing four years of UCSMP mathematics. Journal for 
Research in Mathematics Education, 24(2), 136-158. 

Additional source: 

Hirschhorn, D. B. (1992). Implementation of first four years of 
the UCSMP secondary curriculum (Unpublished doctorai 
dissertation). University of Chicago, IL. 

Thompson, D. R., Senk, S. L., Witonsky, D., Usiskin, Z., & Kaeiey, 
G. (2006). An evaluation of the second edition of UCSMP 
Algebra. Chicago, IL: University of Chicago School Mathemat- 
ics Project. 



Studies that fall outside the High School Math review proto- 
col or do not meet WWC evidence standards 

Davis, J. D., & Shih, J. C. (2007). Secondary options and post- 
secondary expectations: Standards-based mathematics 
programs and student achievement on coilege mathematics 
piacement exams. School Science and Mathematics, 

707(8), 336-346. The study is ineiigible for review because 
it does not use a comparison group design or a singie-case 
design. 

Hedges, L. V., Stodolsky, S., Fiores, P. V., Mathison, D., 

Mathison, S., Sarther, C. S., & Zhang, J. (1988). Formative 
evaluation of USCMP Advanced Aigebra. Advanced Algebra 



10. These instruments demonstrated validity and reliability as discussed in Appendix A2. 
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Appendix 



Appendix A1.1 Study characteristics: Hirschhorn, 1993 



Characteristic 


Description 


Study citation 


Hirschhorn, D. B. (1993). A longitudinal study of students completing four years of UCSMP mafhematics. Journal for Research in Mathematics Education, 24(2), 136-158. 


Participants 


After identifying a set of locations that used four years of (7CS/WP curricula, the researcher focused on three places where both traditional and L/CS/WP curricula were used. 
Every student in the intervention group received four years of L/CS/WP curricula, starting with Transition Mathematics in 7th grade and ending with L/CS/WP Advanced Algebra. 
No student in the control group received any UCSMP courses. 

It is uncertain how students were initially assigned to the different courses, so the researcher matched students on a pretreatment outcome ex post facto. Since the interven- 
tion began in 7th grade, students were matched on their 6th-grade performance on a standardized exam (in 1986).^ The process differed slightly by site. In site A, the only 
available data on 6th-grade academic performance were composite scores for math, reading, and general logic; because baseline equivalence on math achievement alone 
could not be established, this site is excluded from WWC analysis. In sites B and C, students were essentially matched on 6th-grade math scores and then on 6th-grade read- 
ing scores. Since some of the control students in sites B and C had not taken advanced algebra by the end of 10th grade, two comparison groups were formed. We focused 
on the age-based comparison sample, wherein the comparison group was based on all students who started 7th grade at the same time (and thus were in 10th grade at the 
time of the posttest, in 1990).^ The eligible sample for this intervention report was 62 students (31 intervention and 31 comparison): 26 students in site B (13 intervention and 
13 comparison) and 36 students in site C (18 intervention and 18 comparison).^ 

We pooled results across sites B and C because when sites were analyzed separately, all students in the treatment group in site B had the same teacher and class in at least 
one grade, thus generating a confound. 


Setting 


This study took place in three locations that were unspecified due to confidentiality concerns. Sites B and C were affluent suburbs with predominantly White, relatively high- 
performing students (in the 80th to 90th percentile nationally in both reading and math). In site B, students were selected from a single class in the largest of seven feeder 
schools serving the district high school. In site C, students were selected from four classes in three feeder schools to the high school serving the district. 


Intervention 


The intervention students used UCSMP math curricula for four consecutive years, starting in 7th grade and ending in 10th grade. They used Transition Mathematics in 
1986-87, (/CS/WP Algebra in 1987-88, (7CS/WP Geometry in 1988-89, and L/CS/WP Advanced Algebra in 1989-90. 


Comparison 


The comparison students used traditional math curricula that correspond to each of the UCSMP courses, produced by Addison-Wesley, Merrill, Houghton Mifflin, and others. 


Primary outcomes 
and measurement 


The Mathematics Level 1 Achievement Test, Form 3JAC2 (College Board, 1988), a standardized eligible outcome, was measured in 10th grade, near the end of the final 
academic year (April/May 1990). The developer-created math achievement test, administered during the same period, also was considered. For a more detailed description of 
these outcome measures, see Appendix A2. 


Staff/teacher training 


There was no specific staff or teacher training specified. However, at site B, some of the 7th- and 8th-grade teachers were involved with the UCSMP pilot, so they had previ- 
ous experience working with the curricula. 



1 . There are potential concerns with matching a comparison sample based on 6th-grade test scores when the posttest is taken in 10th grade. This is especially problematic when the treatment 
group consists of students who have persisted in the treatment for four consecutive years, which is likely to be a select sample of initial entrants (when even the initial entrants could be a self- 
selected group), compared with a retrospectively chosen comparison group. 

2. The second course-based comparison group used students at the end of advanced algebra. Since that could involve comparing lOth-grade treatment students with llth-grade comparison 
students, we did not include this comparison in the intervention report. 

3. We excluded the 22 students in site A (11 intervention students and 11 comparison students). 
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Appendix A1.2 Study characteristics: Thompson et ai., 2006 



Characteristic 


Description 


Study citation 


Thompson, D. R., Senk, S. L., Witonsky, D., Usiskin, Z., & Kaeley, G. (2006). An evaluation of the second edition of UCSMP Algebra. Chioago, IL: University of Chicago Schooi 
Mathematics Project. 


Participants 


Schoois were recruited for the study by advertising in UCSMP and Nationai Councii of Teachers of Mathematics pubiications. To participate, a schooi needed at ieast four sec- 
tions of the equivaient of an aigebra ciass, at either middie or high schooi, and had to promise to keep ciasses intact for a fuii year. The seiected schoois provided the names of 
at ieast two teachers wiiiing to participate in the study. The study utiiized a matched-pairs design, in which ciasses were matched in the same schooi on the basis of students’ 
math abiiity. When possibie, teachers were randomiy assigned to treatment conditions within each pair; however, iocai conditions did not aiways permit random assignment. 

The treatment group used t/CS/WP Aigebra second edition, whereas the comparison group used either L/CSMP Aigebra first edition or a non-L/CS/WPcomparison textbook. 

The former comparison was not eiigibie for review based on the topic area review protocoi. This review focuses on the three high schoois — X, Y, and Z as iabeied by the 
authors — that compared the UCSMPsecond edition to non-(7CS/WPtextbooks. Anaiysis sampie size was 189 students (98 intervention and 91 comparison): 28 in schooi X 
(14 intervention and 14 comparison), 114 in schooi Y (65 intervention and 49 comparison), and 47 in schooi Z (19 intervention and 28 comparison).^ The majority of these 
students (n=161) were in 9th grade; the rest were enroiied in grades 10-12. 


Setting 


The three high schoois inciuded in the resuits were iocated on the West Coast, in the Northeast, and in the South. Schooi X was a iarge, ethnicaiiy diverse high schooi on the 
West Coast, serving roughiy 2,800 students in grades 9-12 from inner-city and suburban environments; the oniy UCSMPtext previousiy used at the schooi was Geometry. 
Schooi Y was a suburban high schooi in the Northeast, serving 950 students in grades 9-12 from a middie- to upper-middie-ciass socioeconomic popuiation; no previous 
UCSMPtexXs were used at this schooi. Schooi Z was a suburban high schooi of about 2,800 students in grades 9-12, which was iocated in a middie- to upper-middie-ciass 
neighborhood in the South and serves a iarge Hispanic community; no previous UCSMPXexts were used in this schooi. 


Intervention 


Teachers assigned to the treatment condition used (7CS/WP Aigebra (second edition, fieid triai version) during the 1992-93 schooi year. 


Comparison 


The mn- UCSMP ciasses used the foiiowing aigebra texts at the time of the study: Algebra 1: An Incremental Development (Saxon) in Schooi X, Houghton Miffiin’s Algebra: 
Structure and Method Book l\n Schooi Y, and Prentice Haii’s Algebra I'm Schooi Z. 


Primary outcomes 
and measurement 


Shortiy before the end of the schooi year, the teachers administered severai instruments: (1) High Schooi Subjects Tests: Aigebra, (2) a deveioper-created Aigebra Test, and (3) 
a deveioper-created Aigebra Probiem-Soiving and Understanding Test. The iast test was administered in two forms, with haif the students in each ciass randomiy assigned to 
the even form of the test and the other haif assigned to the odd form. For a more detaiied description of these outcome measures, see Appendix A2. 


Staff/teacher training 


No direct in-service was provided to the teachers using (7CS/WP Aigebra either before or during the schooi year. Aithough teachers had a tentative tabie of contents for the 
entire book when the schooi year began, they received the actuai text in three spirai-bound parts: chapters 1-4 at the beginning of the schooi year, chapters 5-8 around 
November, and chapters 9-13 in eariy winter. Additionaiiy, teachers received iesson notes and answers to questions, one chapter at a time, throughout the schooi year. To 
assist with the formative evaiuation, the teachers compieted a chapter evaiuation form after compieting each chapter. These teachers aiso met in Chicago once in the faii and 
again in the spring to give feedback to the deveiopers about the materiais; during these meetings, there were brief opportunities to raise issues reiated to the use of technoiogy 
and the use of reading and group probiem soiving in ciass and to discuss other instructionai concerns. 



1 . We excluded 334 students from the analysis comparing the UCSMP Algebra second edition to the first edition 
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Appendix A2 Outcome measures for the math achievement domain 



Outcome measure 

Mathematics Level I 
Achievement Test, Form 
3JAC2 (College Board, 1988) 

UCSMP Applications Test 



High School Subjects 
Test; Algebra (American 
Testronics, 1988) 

UCSMP Algebra Test 

UCSMP Problem-Solving 
and Understanding Test 



Description 

This 50-question, multiple-choice, standardized achievement instrument pubiished by the Coiiege Board covers geometry and second-year algebra content (as cited in 
Hirschhorn, 1993). The test was graded on the College Board range of 200 to 800 points. On the reliability tests for the samples in the study, the coefficient alpha ranged 
from 0.601 to 0.925 (median of 0.873). 

This 30-question, multiple-choice test was developed by UCSMP for this study as well as for use by future researchers and school personnel (as cited in Hirschhorn, 1993). 
Scientitic calculators were provided and their use was encouraged. Items were selected from the Second Internationa! Mathematics Study {ChauQ & Ruzicka, 1985), the 
Formative Study of UCSMP Advanced Algebra (Hedges et al., 1988), a submittal to the College Board by personal letter in 1988, and some original items. On the reliability 
tests tor the samples in the study, the coetficient alpha ranged from 0.637 to 0.842 (median ot 0.739). 

This 40-question, multiple-choice standardized test (as cited in Thompson et al., 2006) focuses on algebraic concepts. Calculators are not allowed during the test. For 
samples in the study, the Kuder-Richardson KR20 was approximately 0.80. 



This 40-item, multiple-choice test was developed by UCSMP to assess algebraic content in L/C5/WP Algebra as well as topics important in all algebra classes regardless ot 
curriculum (as cited in Thompson et al., 2006). Calculators were permitted during the test. For samples in the study, the KR20 was between 0.81 and 0.83. 

This open-ended problem-solving test was administered in two forms, with half the students in each class randomly assigned to the even form of the test and the other half 
assigned to the odd form (as cited in Thompson et al., 2006). Each form contained four open-ended items, and item-specific rubrics were developed for each item. Each item 
was scored independently and blindly by two raters. Inter-rater reliabilities were as follows: 

Even Form: 78.8% (item 1), 89.7% (item 2), 74.2% (item 3), 93.1% (item 4a), 89.4% (item 4b) 

Odd Form: 79.3% (item 1), 82.9% (item 2), 88.1% (item 3), 92.8% (item 4a), 94.8% (item 4b) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 



Outcome measure 


Study 

sample 




Authors’ findings from the study 

Mean outcome 
(standard deviation)^ 




WWC calculations 




Sample size 
(classes/ 
students) 


UCSMP 

group" 


Comparison 

group’' 


Mean 

difference" 
(UCSMP 
- comparison) 


Effect 

size" 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index" 


Mathematics Level 1 
Achievement 


Grade 10, 
sites B & C 


62 students 


Hirschhorn, 1993" 

492,26 489.03 

(46.39) (78.71) 


3.23 


0.05 


ns 


+2 


UCSMP Applications Grade 10, 

sites B & C 

Average for math achievement (Hirschhorn, 1993] 


62 students 


21.10 

(3.88) 


16.46 

(3.95) 


4.64 


1.17 


Statistically 

significant 


+38 


,10 








0.61 


Statistically 

significant 


+23 








Thompson et al., 2006" 










High School Subjects: Algebra 


Mostly grade 9, 


12 Classes/ 


47,9 


46.0 


1,9 


0.12 


ns 


-h5 




schools X, Y, & Z 


189 students 


(16.3) 


(14.9) 










UCSMP Algebra 


Mostly grade 9, 


12 classes/ 


49.5 


37,3 


12.2 


0,78 


Statistically 


-h28 




schools X, Y, & Z 


189 students 


(16.3) 


(14.9) 






significant 




UCSMP Problem-Solving 


Mostly grade 9, 


12 classes/ 


6.23 


3.39 


2,84 


0,89 


Statistically 


-h31 


and Understanding''^ 


schools X, Y, & Z 


189 students 


(3,69) 


(2.54) 






significant 




Average for math achievement (Thompson et al., 


2006)'" 








0.59 


Statistically 


+22 
















significant 




Domain average for math achievement across all studies'" 








0.60 


na 


+23 



ns = not statistically significant 
na = not applicable 



(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ (continued) 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math achievement domain. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure would indicate that participants 

had more similar outcomes. 

3. The intervention group values are pooled across the eligible sites. 

4. The comparison group values are pooled across the eligible sites. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 

The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Hirschhorn (1993), a correction for multiple comparisons was needed, so the significance levels may differ from those reported in 
the original study. In the case of Thompson et al. (2006), corrections for clustering and multiple comparisons were needed, so the significance levels may differ from those reported in the original 
study. 

10. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 

11. The group values for each condition are pooled across even and odd test forms. 



WWC Intervention Report 



University of Chicago School Mathematics Project 6-12 Curriculum 



July 2011 



11 





Appendix A4 UCSMP rating for the math achievement domain 

The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of math achievement, the WWC rated UCSMP as potentiaiiy positive for high school students. The remaining ratings (mixed, no discernibie 
effects, potentiaiiy negative, and negative) were not considered, as UCSMP was assigned the highest applicable rating. 

Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. Two studies of UCSMP showed a statistically significant positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. No studies of UCSMP showed indeterminate or statistically significant negative effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Although two studies of UCSMP showed statistically significant positive effects, neither met WWC evidence standards for a strong design. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies of UCSMP showed statistically significant negative effects. 



1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schoois 


Sampie size 


Students 


Extent of evidence^ 


Math achievement 


2 


5 




251 


Small 



1 . A rating of “medium to iarge” requires at ieast two studies and two schoois across studies in one domain, and a totai sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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